Individual versus general structured feedback to improve agreement in grant peer review: a randomized controlled trial

Abstract Background Vast sums are distributed based on grant peer review, but studies show that interrater reliability is often low. In this study, we tested the effect of receiving two short individual feedback reports compared to one general report agreement between reviewers. Methods A total 42 reviewers at Norwegian Foundation Dam were randomly assigned receive either a or an report. The group received before start reviews contained information about previous call in which participated. group, reports, review period (based call) and during current call). presented with detailed their scoring committee as whole, both period. main outcomes proportion eligibility assessment average difference scores pairs assessing same proposal. measured 2017 after was provided 2018. Results 2398 paired included analysis. There significant groups absolute whether proposal eligible for funding programme, demonstrating higher rate agreement. no terms score difference. However, regarding remained critically low groups. Conclusions We did not observe changes 2018 different feedback. levels remain major concern research identify contributing factors well development testing interventions increase rates still needed. Trial registration study preregistered OSF.io/n4fq3 .